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ABSTRACT 



A new technique for extracting a hierarchical decomposition 
of a complex video selection for browsing purposes, com- 
bines visual and temporal information to capture the impor- 
tant relations within a scene and between scenes in a video, 
thus allowing the analysis of the underlying story structure 
with no a priori knowledge of the content. A general model 
of hierarchical scene transition graph is applied to an imple- 
mentation for browsing. Video shots are first identified and 
a collection of key frames is used to represent each video 
segment. These collections are then classified according to 
gross visual information. A platform is built on which the 
video is presented as directed graphs to the user, with each 
category of video shots represented by a node and each edge 
denoting a temporal relationship between categories. The 
analysis and processing of video is carried out directly on the 
compressed videos. Preliminary tests show that the narrative 
structure of a video selection can be effectively captured 
using this technique. 

16 Claims, 13 Drawing Sheets 
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METHOD AND APPARATUS FOR VIDEO 
BROWSING BASED ON CONTENT AND 
STRUCTURE 

This is a continuation of application Ser. No. 08/382,877, 
filed Feb. 3, 1995 now U.S. Pat. No. 5,708,767. 

FIELD OF THE INVENTION 

The present invention relates generally to methods and 
apparatus for browsing of video material, and more specifi- 
cally to automating the browsing process. 

BACKGROUND OF THE INVENTION 

The ability to browse through a large amount of video 
material to find the relevant clips is extremely important in 
many video applications. In interactive TV and pay-per- 
view systems, customers want to see sections of programs 
before renting them. While prepared trailers may suffice to 
publicize major motion pictures, episodic television, sports, 
and other programs will probably require browsers to let the 
customer find the program of interest. In the scholarly 
domain, digital libraries will collect and disseminate moving 
image material. Many scholars, including political scientists, 
psychologists and historians study moving images as pri- 
mary source material. They require browsers to help them 
find the material of interest and to analyze the material. 
Browsing is even more important for video than for text- 
based libraries because certain aspects of video are hard to 
synopsize. 

Early browsers were developed for video production and 
as front ends for video databases. Today's standard tech- 
nique for browsing is story board browsing in which the 
video information is condensed into meaningful snapshots 
representing shots while significant portions of the audio can 
be heard. One known browser divides the sequence into 
equal length segments and denotes the first frame of each 
segment as its key frame. This browser does not detect scene 
transitions and it may both miss important information or 
display repetitive frames. Another known browser stacks 
every frame of the sequence and provides the user with rich 
information regarding the camera and object motions. 
However, a scholar using a digital video library or a cus- 
tomer of a pay-per-view system is more interested in the 
contents (who, what, where) than how the camera was used 
during the recording. 

A first step toward content-based browsing is to classify 
a long video sequence into story units, based on its content. 
Scene change detection (also called temporal segmentation 
of video) gives sufificient indication of when a new shot 
starts and ends. Scene change detection algorithms, an 
algorithm to detect scene transitions using the DCT coeffi- 
cients of an encoded image, and algorithms to identify both 
abrupt and gradual scene transitions using the DC coeffi- 
cients of an encoded video sequence are known in the art. 

Beyond temporal segmentation of video, one known 
browser uses R frames (representative frames) to organize 
the visual contents of the video clips. Rframes may be 
grouped according to various criteria to aid the user in 
identifying the desired material: the user can select a key 
frame, and the system then uses various criteria to search for 
similar key frames and present them to the user as a group. 
The user could search representative frames from the 
groups, rather than the complete set of key frames, to 
identify scenes of interest. It is known to use a language- 
based model to match the incoming video sequence with the 
expected grammatical elements of a news broadcast, and to 
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use a priori models of the expected content of the video clip 
to parse the clip. 

In reality, many moving image documents have story 
structures which are reflected in the visual content. Note that 

5 a complete moving image document is referred to herein as 
a video clip. The fundamental unit of the production of video 
is the shot, which captures continuous action. A scene is 
usually composed of a small number of interrelated shots 
that are unified by location or dramatic incident. Feature 

10 films are typically divided into three acts, each of which 
consists of about a half-dozen scenes. The act-scene-shot 
decomposition forms a hierarchy for understanding the 
story. News footage also has a similar structure: a news 
program is divided into stories, each of which typically starts 

15 with a common visual cue signaling the start of a new story, 
and with each story containing several shots and perhaps 
multiple scenes. Thus, at the lower levels of the hierarchy, a 
scene may consist of alternating shots of the two main 
characters or an interviewer and interviewee. At the higher 

20 levels of abstraction, an act in a movie or a story in a news 
show may be signaled by a visual motif, such as an estab- 
lishing shot of the locale or a shot of the anchor in front of 
a title card. 

In view of such underlying structures, a video browser 
25 should allow the user to first identify the scenes taking place 
at that location using visual information, select the scene 
desired using temporal information, and similarly navigate 
through the various shots in the scene using both visual and 
temporal information. Thus there is a real need to identify 
30 both visual and temporal relationships to allow the user to 
recognize the underlying story structure and navigate to the 
desired point in the video. 

SUMMARY OF THE INVENTION 

35 In one embodiment of the invention, the story structure is 
modeled with a hierarchical scene transition graph, and the 
scenic structure is extracted using visual and temporal 
information with no a priori knowledge of the content — the 
structure is discovered automatically. A hierarchical scene 

aq transition graph reflects the decomposition of the video into 
acts, scenes and shots. Such a hierarchical view of the video 
provides an effective means for browsing the video content, 
since long sequences of related shots can be telescoped into 
a small number of key frames which represent the repeatedly 

45 appearing shots in the scene. In addition, the analysis and 
processing of video is carried out directly on videos com- 
pressed using common compression standards such as 
Motion JPEG and MPEG. 

The present inventive system presents the content- 

50 structure of the video in a way that appeals to basic human 
understanding of the visual material in video. Also, a frame- 
work is provided on which the navigation in video databases 
can be facilitated both by machine automation and user 
interaction. 

55 BRIEF DESCRIPTION OF THE DRAWINGS 

Various embodiments of the invention are described and 
illustrated herein with reference to the accompanying 
drawings, in which like items are identified by the same 
60 reference designation, wherein: 

FIG. 1 is a block diagram and flowchart for one embodi- 
ment of the invention; 

FIG. 2 shows video shots of the 1992 Democratic Con- 
vention in a sequence based upon temporal order; 
65 FIG. 3a shows clustering results from video shots of the 
1992 Democratic Convention for one embodiment of the 
invention; 
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FIG. 3b shows clustering results from a News Report, for Property (2) deals with grouping of shots at the lowest 

an embodiment of the invention; level of hierarchy. The collection f shots is partitioned into 

FIG. 4 shows an initial scene transition graph of the 1992 nodes of G <>; eacn Qode represents a cluster of shots, which 

Democratic Convention video sequence, from an embodi- a *"e considered a scene in the general sense. A directed edge 

ment of the invention; 5 is drawn from node U to W if there is a shot represented by 

FIG. 5 shows an initial scene transition graph of the News n0 ^ " thai immediately precedes some shot represented by 

Report sequence, for an embodiment of the invention; nod * ™ ? urther &°^ m f ""° 0,her levek 0 * e ^ erar <; h y 

i. , . . , , is denned in a similar fashion in property (3). The edge 

FIG. 6. shows a top-level scene transition graph of the relationsnips induced by lemporal precedence at leve , 0 is 

1992 Democratic Convention video sequence, for one w p rese rved as one moves up the hierarchy. 

embodiment of the invention; 

FIG. 7 shows a top-level scene transition graph of the EXAMPLES 
News Report video sequence, for an embodiment of the 

invention; Example 1 

FIG. 8 shows a block schematic diagram of a system for 15 t d . fQ , 

implementing the methods of various embodiments of the iree ^presentation ot bnots 
invention; and 

FIGS. 9, 10, 11 and 12 show flowcharts for various f 0 partitions {s,} into v 0il -{si v Qa -{$ L + u .... S2L }, . 

embodiments of the invention, respectively. • ■ . (4) 

20 

DETAILED DESCRIPTION OF THE partitions {v w } into v 1(1 -{v 0>1 , . . . ,v 0tL ), v li2 ={v 0X+1 , 



INVENTION 



>L ) . . . , and so on. (5) 



o t* ■* ■ /** l. j wj r» • This is a hierarchical organization in time of the collection 

Scene Transition Graphs and Video Browsing * u , A * ,u i .11 u jt/ , t 

y 6 of shots. At the lowest level, each node V 0 j . represents L 

This section first defines the general model of hierarchical 25 shots; a directed edge connects V 0ii to V 0 l+J , giving a 

scene transition graphs, presents the notations and defini- directed path as the structure of the graph at this level. At the 

tions of the terms used herein, and finally provides examples next level, each node V v represents L nodes of G 0 . Such a 

that the model represents. tree hierarchy permits a user to have a coarse-to-fine view of 

the entire video sequences. 

The temporal relation defined by the edge in this example 



Notation and definitions 30 



A shot is defined as a continuous sequence of actions, as is verv simple. The relations between shots in a video tends 
a fundamental unit in a video sequence. The ith shot is to be more complicated. The next example defined {F t j to 
denoted by: allow a better representation of content -based dependency 



between the shots. 

Example 2 



Go-(V 0 , Bo, Fq) (2) 



where f ; is the jth frame and b, is the beginning time of the 

shot and e ( . is the ending time of the shot. The relation Directed graph representation of shots 
b 1 <e 1 <b 2 <e 2 <b 3 < ... is also assumed. In addition, we call 

a collection of one or more interrelated shots a scene. Many 4Q is one level of hierarchy, i.e. H-l. F 0 partitions {sj 

criteria can be imposed to determine if the shots are related int0 v o,i> v o,2> ■ • • » such thal nodes in eacn v o, ( - are 

or nol sufficiently similar, according to some similarity measured 

A Hierarchical Scene Transition Graph (HSTG) on {s,-} is in terms of low level ™ ion indices such as colors > sha P es > 

a collection of directed graphs ctc - 

45 In this case, shots that are similar to each other are 

, C j //-i clustered together. Relations between clusters are governed 

1 m 0 by temporal ordering of shots within the two clusters. A 

simple example would be a scene of conversation between 

with H denoting the total number of levels in the hierarchy, two persons; the camera alternates between shots of each 

which has the following properties: $Q person. The graph G 0 consists of two nodes V 01 and V 0 2 ; 

(V 0 j, V 0>2 ) and (V 0 2 , V 0 tl ) are both members of E 0 . 
It should be noted that many implementations can fit in 

where V 0 -{ V 0 J is the node set and E 0 is the edge set. F 0 this model. One possible browsing system embodiment is 

is a mapping that partitions {sj into V 0 lt V 02 , . . . , the shown below under such a generalized framework in the 

members of V 0 . For given U, W, in V 0 , (U,W) is a member 55 following sections. Extensions of this model to include more 

of E 0 if there exists some Sj in U and and s m in W such that levels of the hierarchy is possible by using semantic 

m=l+l. information, derived from further processing or from human 

For i=l,2, . . . , H-l: interaction, as well as temporal information. 

g ( -(v„e ( , F,). (3) 6o A Method For Video Browsing For One 

Each V,. and E £ is the node set and edge set respectively. F ( Embodiment 

partitions V M into V t l , V l 2 , . . . , the members of V,.. For A block diagram approach for one embodiment of the 

given U, W, in V ( -, (U,W) is a member of E; if there exists invention is shown in FIG. 1. The browsing process is 

some v, in U and and v m in W such that (v„v m ,) is a member automated to extract a hierarchical decomposition of a 

of E M . 65 complex video selection in four steps: the identification of 

The hierarchy is uniquely defined by the action of {F,} on video shots, the clustering of video shots of similar visual 

the collections of shots {s,-}. contents, the presentation of the content and structure to the 
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users via the scene transition graph, and finally the hierar- 
chical organization of the graph structure. 

All processing aspects of this system embodiment are 
performed on reduced data called DC-sequence derived 
directly from compressed video using Motion JPEG or 5 
MPEG. A DC-sequence is a collection of DC images, each 
of which is formed by blockwise averaging of an original 
image. In the case of a discrete cosine transform (DCT) 
based compression scheme, as in JPEG and I-frame of 
MPEG, the DC image is the collection of DC coefficients 10 
scaled by 8. The DC images in P and B frames of MPEG 
compressed sequences can be derived in terms of the DCT 
coefficients of the anchor frames, thus permitting the extrac- 
tion of DC sequences directly from MPEG sequences. No 
full frame decompression is necessary. This approach has 15 
the advantage of computational efficiency because only a 
very small percentage of the entire video sequence is used. 
In addition, the use of DC sequence for processing permits 
a uniform framework for scene and content analysis on 
Motion JPEG and MPEG compressed video embodied in the 20 
system. 

The identification of video shots are achieved by scene 
change detection schemes which give the start and end of 
each shot. A detection algorithm as described by B. L. Yeo 
and B. Liu, in "Rapid scene analysis on compressed videos", 25 
an unpublished paper submitted to IEEE Transactions on 
Circuits and Systems for Video Technology, is used, and the 
Yeo and Liu paper is incorporated herein by reference in its 
entirety to the extent it does not conflict herewith. The 
algorithm handles with high accuracy both abrupt changes 30 
and special effects like gradual transitions which are com- 
mon in the real-world video. The shots that exhibit visual, 
spatial and temporal similarities are then clustered into 
scenes, with each scene containing one or more shots of 
similar contents. Each such cluster is a basic note in the 35 
scene transition graph. From the clustering results and the 
temporal information associated with each shot, the system 
proceeds to build the graphs, with nodes representing scenes 
and edges representing the progress of the story from one 
scene to the next. The nodes capture the core contents of the 40 
video while the edges capture its structure. The browsing 
approach thus is based on both content and structure of a 
complex video selection. While the primitive attributes of 
the shots contribute the major clustering criteria at the initial 
stage of the scene transition graph construction, hierarchy of 45 
the graph permits further organization of the video for 
browsing purposes. At each level a different criterion is 
imposed. The lower levels of the hierarchy can be based 
upon visual cues while the upper levels allows criteria that 
reflect semantic information associated. This hierarchical 50 
organization offers a browsing structure that closely 
resembles human perception and understanding. 

The present method offers a better organization of video 
contents than existing storyboard browsing schemes in 55 
facilitating the browsing process. In addition, no a priori 
models are assumed. It also allows user interaction to 
supplement machine automation in the course of analysis. 
This is very crucial in video browsing as perception and 
understanding of contents varies with individuals and no 6Q 
existing automation schemes have yet to achieve precise 
accuracies. 

Clustering of Video Shots 

The temporal segmentation process of the video clip 65 
produces a collection of shots {s,}, arranged in a linear array 
fashion according to their temporal orderings, as shown in 
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FIG. 2. Each shot is visually represented by its representa- 
tive DC image. This presentation format relieves the user 
from the need to watch the entire video during browsing. 
However, a typical one hour program can produce hundreds 
of different shots. In such cases, presenting shots in a one 
dimensional image array does not offer the users an effective 
and efficient means to browse, navigate and search for any 
particular video segments. This presents a greater challenge 
for the user if the user has never watched the video in its 
entirety, and has no idea where in the time array the search 
should start. 

A hierarchical scene transition graph offers a better orga- 
nization of video contents than existing story-board brows- 
ing schemes and facilitates the browsing process. Clustering 
of video shots is the first step toward the building of the 
graph. The sequence {sj is grouped into clusters, denoted 
by ^o,i> V 0 2 , ... by measures of similarity. Verification of 
the clustering results is done through user interaction. Some 
degree of user verification is necessary to allow flexibility 
and because no a priori model is assumed for the content- 
structure of the video under investigation. 

Similarity of shots 

Low level vision analyses operated on video frames 
achieve reasonably good results for the measurement of 
similarity (or dissimilarity) of different shots. Similarity 
measures based on image attributes such as color, spatial 
correlation and shape can distinguish different shots to a 
significant degree, even when operated on much reduced 
images as the DC images. Both color and simple shape 
information are used to measure similarity of the shots. 

Color 

Most of the video shots encountered everyday are shots of 
real scenes, real people and realistic motions. It is rare that 
two shots of very different contents will have very similar 
visual colors. Color is an effective means to distinguish 
different shots. Swain and Ballard in their paper "Color 
Indexing" ^International Journal of Computer Vision, Vol. 7, 
pp. 11-32, 1991, pointed out that "although geometrical 
cues may be the most reliable for object identity in many 
cases, this may not be true for routine behavior: in such 
behavior wherein familiar objects are interacted with 
repeatedly, color may be a far more efficient indexing 
feature." The present inventors have confirmed this, and 
have found that in video with underlying story structures, 
like news broadcasts and night shows, where familiar scenes 
and people appeared and interacted repeatedly, color is far 
more effective and superior when used to classify video 
shots than moment invariant. 

The present system adapts the histogram intersection 
method from Swain and Ballard, as cited above, to measure 
the similarity of two images based on color histograms. A 
color histogram of an image is obtained by dividing a color 
space (e.g. RGB) into discrete image colors (called bins) and 
counting the number of times each discrete color appears by 
traversing every pixel in the image. 

Given two images f,. and fy, and their histograms P and I', 
each containing n bins, \\ denotes the value of I 1 in the kth 
bin. The intersection of the histograms is defined to be: 

This gives an indication of the number of pixels that have 
similar colors in both images. It can be normalized to a 
match value between 0 and 1 by the following: 
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s{m-^— n . w ^ 

V The performance using the luminance projections is com- 
parable to that of using full correlation. Thus equation (11) 

The present system uses mismatch value, D(T, lO-l-S ^ use d to represent the measure of correlation in two 

(I 1 ,!^) to indicate dissimilarity between the two images images, 
instead of the match value defined above. 

In the present implementation, the inventors found that a Temporal variations in a video shot 

8x8x4 discretization of the RGB color space is adequate to 10 ., . . . 

distinguish DC images of size 40x30 pixel square. u S,nce • v «*° sh °' "TT* m ™* lm J^ S ™ U , anty 



Shape 



between the shots needs to be defined beyond that of planar 



The present system uses as another measure of similarity ™*B«?- ! f onl y a , representative image is chosen for a video 

between two images the twonlimensional moment invariant sl \ ot ' »' 15 ver y that ,herc arc two shots J*«* " 6 

of the luminance. However, the inventors discovered that the 15 ' nd r eed ^ much L the same m cont< l nt but a ' e J ud e ed to be 

order of magnitudes in different moment invariant vary dlff6r ? nt becau » ,i he representative frames chosen for these 

4l • „ _ , ( . rtFfiw( Ml -„„ rt ^„J* two shots are different. For example, given the following 

greatly: in many examples the ratio or first moment invariant , , « , ■ . & , . % 

to the third or fourth moment invariant can vary by several ! wo A sho u tS: 11,(5 flrSt * hot StartS W,h a ?°nwd-m view of 

orders of magnitude. ' camera th ? Q f 00 ™ out , t0 lnchlde ' he 

By using the Euclidean distance of the respective moment 20 * econd shot starts with the camera focusing on Ms.B, then 

invariant as a measure of dissimilarity between two images, lt . zo ° ms ° ut l ° delude Mr.A. One may think theses shots are 

such distance will most likely be dominated by the moment " m,lar h »fP"» often m the anchor room during news 

invariant with the highest order of magnitude. The purpose broadcasts), yet if the first image from each shot is taken as 

of using more than one moment invariant for shape match- < he !^? U J£ ^a' re P r 1 f senta ' ,ve , ,ma S es can 

ing will then be totally defeated. The present system 25 be drastically different dependmg on the sim.lanty measures 

attempts to scale the moment invariant to comparable orders use ' 

of magnitudes, or to weigh different invariants by different In a video snot where object and camera motions are 

weights, have not produced agreeable matching results. In prominent, a representative image is not sufficient for the 

fact, the inventors determined that clustering based on faithful analysis of the image set it represents. Yet it is very 

matching by Euclidean distance on the moment invariant 30 computational burdensome to include every frame in the 

gives inferior results in comparison to those produced using snot for processing purposes, as much of the temporal 

color only. information is redundant. It is important to balance the two 

goals: to preserve as much of the temporal variations as 

Correlation of images possible and to reduce the computing load needed to process 

The inventors discovered that measuring correlation 3S many video frames in a given shot. In the present system, the 

between two small images (even the DC images) does give inventors chose a good but nevertheless greatly reduced 

a very good indication of similarity (it is actually dissimi- representative set of frames to represent a video shot, 

larity in the definition below) in these images. By using the However, it should be noted that while a representative 

sum of absolute difference, the correlation between two image is used to represent a shot in the presentation of the 

images, i m and f M is commonly computed by: 40 visual results, the analysis and clustering is not confined to 

only one such representative image. 

/ K (8) 

/-i Clustering Algorithms 

The correlation is known to be very sensitive to transla- A5 A proximity matrix for the video shots are built based on 

tion of objects in the images. However, when applied to the the dissimilarity measures defined for color, shape and 

much reduced images, effects due to object translation correlation for their representative image sets. Each entry of 

appear to lessen to a great degree. the matrix is a proximinity index. Here the inventors fol- 

The inventors found that correlation measures can achieve lowed the definitions in a paper by Jain and Dubes, "Algo- 

clustering results as good as those done by color. To further 5Q rithms for Clustering Data", Prentice Hall, 1988, to define 

reduce the storage space needed and increase the computa- the proximity index on dissimilarity: A proximity index 

tional efficiency, the inventors devised a simple way to between the ith and kth patterns (s, and s^) is denoted by 

measure correlation using the luminance projection defined d(ijc) and must satisfy the following three properties: 
as follows. For a given image f^Q^j^l^, . . . J and 

k»l,2, . , . , K, the luminance projection for the 1th row is: d(M>o, for ail i (12) 

j (9) d(i*)=d(k,i),forall(i,k) (13) 

JVCO- s LumW-a/)}; 

;=3 d(i,k)&0,forall (i,k) (14) 

and luminance projection for the 1th column is: For two shots s . and ^ ^ proximity index can be D(i,k) 

60 for color, e(ijc) for correlation and U(i,k) for shape, or any 
p e m (l) « z Lum combination of the three. Note that the three measures of 

dissimilarity are not transitive. 

This is an array of size K+J and does not require the JxK The proximity matrix serves as the only input to a 

storage for the whole image for correlation calculation in the clustering algorithm. A clustering is a type of classification 

later stages. To test the similarity of images f m and f„, the 65 imposed on a finite set of objects, whose relationship to each 

sum of absolute difference of the row and column projec- other is represented by the proximity matrix in which rows 

tions is used as follows: and columns correspond to objects. 
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To allow for user interaction, the inventors chose hierar- The scene transition graph makes use of the temporal 

chical clustering over partitional clustering (the latter gives relations to mark the edges of the directed graph, 

a single partition, and the former is a sequence of nested Nevertheless, this structure can also be applied to the spatial 

partitional classifications). The present system algorithm relations within an image or spatio-temporal relations within 

first groups the pair of shots that are most similar together, 5 a shot. In that case, one can further enhance the hierarchy of 

and then proceeds to group other shots by their proximity me g ra P n t0 represent an even better understanding of video 

values. The proximity values between clusters are updated in and add t0 the machine classification capability, 

the process of grouping. However, any clustering algorithms Before aU the are resolved and properly defined, 

that give partitions similar to human classifications based on one needs to investigate closely how the users interact with 

visual characteristics of the video shots will do for this 10 the browsing sys^ 

purpose. In addition, only a good (not perfect) clustering °P tl0 » for * e u ^ rs t0 S rou P hwiarthicaUy the nodes of the 

r , r , j ,i • .... . \ ,, r u r & graph based on their subjective criteria. At the same time, by 

scheme can be used as the initial step to the building of scene ^ m0 nitoring what criteria they impose at different 

transition graph. The present system allows users to i eve ^ and how these subjecUve criteria can be decomposed 

re-classify the ; mismatches. Furthermore, underclassification into more concrcte cntities for machinc automation( thc 

is always preferred to overclassification to better accommo- is hierarchy for the graph can be improved. In FIGS. 6 and 7, 

date the hierarchical scene transition graph. This means that potentially useful top level scene transition graphs of the 

it is preferred to have a shot left as a single cluster than to respective video sequences of FIGS. 3a and 36, respectively, 

have it grouped into other clusters not in close match. arc sn own. 

The user is given the flexibility to interactively select the In F IG. 4, an initial scene transition graph of the 1992 
number of clusters desired or to set caps on the dissimilarity 20 Democratic Convention video sequence is shown in the 
values between individual shots allowed in a cluster. In test central porticm of the figure A highlighted node selected by 
trials of the present system, after the initial shot partitions, a ^ ^ illustrated in the center of the initial scene transition 
the user only needs to slightly adjust the knobs to change graph v ia a bold arrow and darkening of the edges of the 
these partitions to yield satisfactory results, often with less highlighted node. The top window shows the video shots 
than four such trials. FIGS. 3a and 36 show clustering results 25 represented by the highlighted node. The bottom window in 
on two sequences: a 16-minute Democratic Convention the figure shows the video shots of the scene transition graph 
video sequence, and a News Report, respectively. The arranged in temporal order. Similarly in FIG. 5, the upper 
directed scene transition graph is laid out using the algo- and pr i ma ry portion shows an initial scene transition graph 
rithms disclosed by Laszlo Szirmay-Kalos, in a paper 0 f a N ews Report video sequence. A lower center window 
"Dynamic layout algorithm to display general graphs 7 ', in 30 shows the video shots of the scene transition graph arranged 
Graphics Gems IV, pp. 505-517, Academic Press, Boston, m the temporal order. The bottom window shows the video 
1994. FIGS. 4 and 5 show the sample interface and graph shots associated with the highlight node, 
layout of the two above-mentioned video sequences, based with further reference to FIG. 6, a top-level scene tran- 
on the results in FIGS. 3a and 36, respectively. Each node sition graph of the 1992 Democratic Convention video 
represents a collection of shots, clustered by the method 35 sequence is shown. The lower window shows the contents or 
described above. For simplicity, only one frame is used to video shots of the highlighted node. The nodes of the scene 
represent the collection of shots. A means is also provided transition graph are typically representative of complete 
for the users to re-arrange the nodes, group the nodes scene changes relative to the nodes of the initial scene 
together to form further clusters, and ungroup some shots transition graph of FIG. 4. Similar comments apply to FIG. 
from a cluster. This enables the user to organize the graphs 40 7 relative to FIG. 5, for the News Report video sequence, 
differently to get a better understanding of the overall a block schematic diagram of a hardware system pro- 
structures, grammed for processing various methods of the invention is 

Image attributes have served as the measurement of shown in FIG, 8, Flowcharts for the programming for 

similarity between video shots at the low levels of the scene selected ones of the steps of FIG. 1 are shown in the 

transition graph hierarchy. In tests of the present method and 45 flowcharts of FIGS. 9, 10, 11 and 12. 
system, the matching of shots based on primitive visual 

characteristics such as color and shape resembles the process Conclusions 

in which the users tend to classify these video shots, when The present method provides a general framework of 

they have no prior knowledge of the video sequence given hierarchical scene transition graphs for video browsing. Also 

to them. However, in addition to color and shape, the users so presented is a new browsing method through the identifica- 

are capable of recognizing the same people (in different tion of video shots, the clustering of video shots by 

backgrounds, clothes, and under varying lighting similarity, and the presentation of the content and structure 

conditions) in different shots, and classify these shots to be to the users via the scene transition graph. This multi-step 

in the same cluster. Further classification and grouping is analysis of the video clip reveals the relationship between 

possible after the users have acquired more understanding of 55 visual and temporal information in the clip. The combination 

the video sequences. captures the important relationships within a scene and 

This suggests that automatic clustering schemes for the between scenes in a video, thus allowing the analysis of the 

scene transition graph building can be made at multiple underlying story structure without any a priori knowledge, 

levels. At each level, a different criterion is imposed. The The feasibility of this framework has been demonstrated 

inventors considered that vision techniques are the keys in 60 by an implementation that automates the process of shot 

the lower levels: image attributes contribute the clustering identification, clustering of similar shots, and construction of 

criterion at the bottom level, image segmentation (e.g. scene transition graph. All processing are performed on 

foregrounds and backgrounds) and object recognition can be DC-sequences that are extracted directly from compressed 

the next level of classification. In the top levels of the MPEG and Motion JPEG sequences. The greatly reduced 

hierarchy, subgraph properties and temporal structures, such 65 data size permits fast computation. For example, the clus- 

as discovering repeated self-loops and subgraph tering on the 16-minute Democratic Convention 1992 

isomorphism, can be explored to further condense the graph. sequence of FIG. 3a is performed in seconds on an SGI Indy. 
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The scene transition graph can be further refined hierar- 7. The video browser of claim 1, further comprising 

chically. Each level has the flexibility of being constructed means for grouping video shots of at least two of the 

via different criteria to reflect both visual and semantic plurality of nodes to form an additional node, 

structure that closely resemble human perception and under- g ^ video browser of claim if fu rlher comprising 

standing. Thus the scene transition graph provides users with s , . . • , , , f i a „ M Mfl «nu 

, .? i r i j means tor ungroupmg video snots rrom at least one ol the 

a tool for better understanding and use of complex videos. , Vl c . \ r , , 

b r plurality of nodes to form an additional node. 

Although various embodiments of the invention are 9 A mcthod for nti video shots for br0 wsing, 

shown and descnbed herein, they are not meant to be me ste s 0 f*. 

limiting. Those of skill in the art may recognize certain comprising e s eps o . 

modifications to these embodiments, which modifications 10 defining a plurality of nodes, each of the plurality of nodes 

are meant to be covered by the spirit and scope of the representing a collection of the video shots; 

appended claims. defining at least one temporal relationship between at 

What is claimed is: l eas t first and second ones of the plurality of nodes; and 

1. A video browser comprising: displaying graphical representations of the plurality of 
node defining means for defining a plurality of nodes, nodes and the at least one temporal relationship. 

each of the plurality of nodes representing a collecting 10 ^ melhod of daim 9 wherein the videQ sho(s of 

of video shots; cach of thc p i ura ij ty 0 f nodes are interrelated by at least one 

temporal relationship defining means for defining at least of ^lor, sp atia i correlation, and shape, 

one temporal relationship between at least first and 2Q n ^ melhod of claim 9 where in the video shots of 

second ones of the plurality of nodes; and each of the p i urality of nodes are arra nged in temporal order. 

a display for displaying graphical representations of the 12. The method of claim 9, wherein the graphical 

plurality of nodes and the at least one temporal rela- representation(s) of the temporal relationship is an edge, 

tionship. 13 Th e method of claim 9, wherein the graphical repre- 

2. The video browser of claim 1, wherein the video shots M sentations of the plurality of nodes aQd ^ at least one 
of each of the plurality of nodes are interrelated by at least , e L relationshi are dis j a d ^ a hi6rarchical mm 
one of color, spatial correlation and shape. transition ah 

3. The video browser of claim 1, wherein the video shots ^ »l j r 1 • i-» L •*!.«.■ . . . 
r i_ r.i. 1 v» r j j ■ * i 14. The method of claim 13, wherein the hierarchical 

of each of the plurality of nodes are arranged in temporal t . , , ' . r , 

Qrc j er or scene transition graph includes a plurality of levels and 

4. Tbc video browser of claim. 1, wherein the graphical 3 ° wherein each of the levels can be constructed according to 
representation of the temporal relationship is an edge. different criteria. 

5. The video browser of claim 1, wherein the graphical 15 ^ method of claim 9, further comprising the step of 
representations of the plurality of nodes and the at least one grouping video shots of at least two of the plurality of nodes 
temporal relationship are displayed as a hierarchical scene 35 t0 f orm an additional node. 

transition graph. 16- The method of claim 9, further comprising the step of 

6. The video browser of claim 5, wherein the hierarchical ungrouping video shots from at least one of the plurality of 
scene transition graph includes a plurality of levels and nodes to form an additional node. 

wherein each of the levels can be constructed according to 

different criteria, ***** 
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